Fourier Transforms by lkasselm · Pull Request #1403 · parthenon-hpc-lab/parthenon

lkasselm · 2026-06-04T14:46:47Z

PR Summary

This PR adds infrastructure for performing distributed Fast Fourier Transforms (FFTs) on uniform meshes in Parthenon. It introduces two new utility classes, a generic spectral output type, and a self-contained example demonstrating correct usage.

Most of this PR (including the documentation) was created in collaboration with LLM chatbots. Claude code refactored the spectrum calculation into its own utility function and created the regression test.

PR Checklist

… lkasselm/fft

…build-in

…nter interface.

pgrete · 2026-07-10T17:41:23Z

I just pushed some changes to the PR, in particular

make the machinery work with Real=float
separating the precision in the spectra calculation (so that these are always calculated in SpecReal = double, which is important for the big boxes in single precision output)
Fixed compilation/linking when heffte is externally used
added test case to the CalcSpectrum function (now within the utils::fft namespace), which implicitly tests the Hermitian prefactor in calculating the power
added an interface to directly pass a data pointer (rather than field string) to CalcSpec
added a more performance way for FFTs on GPU using heffte options

This PR should go in after #1412 (as it already contains those changes).

@Yurlungur would be great if you could review (briefly again) so that we get the second approval.

Yurlungur

LGTM. Minor nitpicks remaining but they're nonblocking

Yurlungur · 2026-07-10T20:40:28Z

+       int low[3];    // lower bound in each dimension
+       int high[3];   // upper bound in each dimension
+       int size[3];   // size in each dimension: high - low + 1
+   };


Can we re-use the RegionSize struct?

Regionsize currently only holds effectively the size aspect (plus position related Real info), but we need the lower and upper index bounds here.

Yurlungur · 2026-07-10T21:20:42Z

 add_subdirectory(sparse_advection)
+if(PARTHENON_ENABLE_FFT)
+  add_subdirectory(fourier_transform)
+endif()


missing newline

Yurlungur · 2026-07-10T21:21:56Z

+class SpectralOutput : public OutputType {
+ public:
+  explicit SpectralOutput(const OutputParameters &oparams) : OutputType(oparams) {}
+  void WriteOutputFile(Mesh *pm, ParameterInput *pin, SimTime *tm,
+                       const SignalHandler::OutputSignal signal) override;
+};


should this also be guarded behind the enable FFT macro?

I think this can stay (so that there's no accidental overwrite by another SpectralOutput type)

Yurlungur · 2026-07-10T21:22:14Z

+//! \fn void SpectralOutput::WriteOutputFile()
+//  \brief Writes a spectrum output file
+
+void SpectralOutput::WriteOutputFile(Mesh *pm, ParameterInput *pin, SimTime *tm,


see above question about macro

This should be safe as the entire file is only selectively included through the macro in CMakeLists.txt

Yurlungur · 2026-07-10T21:22:31Z

+      fout << std::format("{:d} {:.15e} {:.15e} {:.15e}\n", i, spectra_h(i, 0),
+                          spectra_h(i, 1), spectra_h(i, 2));


pgrete · 2026-07-16T21:43:49Z

Current CI image doesn't ship rocfft (required by heffte).
I fixed this in #1414, which now needs to go in before we can merge this PR.

pgrete · 2026-07-16T21:46:20Z

@lkasselm the fail in the CUDA test seems legit:

[21](https://github.com/parthenon-hpc-lab/parthenon/actions/runs/29504003750/job/87639925037?pr=1403#step:9:222)
Initialized UniformGridHelper
Mesh layout:
Rank 0 local mesh box: low = (0, 0, 0), high = (15, 15, 15), size = (16, 16, 16)
*** The MPI_Comm_rank() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[67e8270543fe:30505] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

see https://github.com/parthenon-hpc-lab/parthenon/actions/runs/29504003750/job/87639925037?pr=1403

Do you want to take a look at this?
We might need to check that MPI (communicators) are properly initialized (in proper order) and, similarly, also closed cleanly on termination.

pgrete · 2026-07-28T15:51:10Z

@lkasselm the fail in the CUDA test seems legit:
[21](https://github.com/parthenon-hpc-lab/parthenon/actions/runs/29504003750/job/87639925037?pr=1403#step:9:222)
Initialized UniformGridHelper
Mesh layout:
Rank 0 local mesh box: low = (0, 0, 0), high = (15, 15, 15), size = (16, 16, 16)
*** The MPI_Comm_rank() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[67e8270543fe:30505] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
see https://github.com/parthenon-hpc-lab/parthenon/actions/runs/29504003750/job/87639925037?pr=1403

Do you want to take a look at this? We might need to check that MPI (communicators) are properly initialized (in proper order) and, similarly, also closed cleanly on termination.

Argh, it's the serial test that fails, so no MPI...
I'll fix this.

pgrete · 2026-07-29T14:20:20Z

Wait what?!
How did this merge go through with one test failing?!

pgrete and others added 30 commits April 15, 2024 15:50

WIP Read/Write Params

f9373e8

Make ReadParams private member

96a3f4c

Merge remote-tracking branch 'origin/develop' into pgrete/pmd-output

61306a0

Fix root level in output

56976a8

Move to mesh per record standard for writing

6199843

Merge remote-tracking branch 'origin/develop' into pgrete/pmd-output

9fb9f68

Allow for 2D and 3D output. Fix single dataset reset.

684b7ab

Fix logical loc

251c6ea

Rename opmd files

03f80c7

Separate common calls to chunks and names

890fffe

Reuse shared chunk and name for restarts

04359d3

Add regression test

2b89659

Somewhat make restarts working

804e60d

Fix order of arguments for correct flush

a436f55

Merge remote-tracking branch 'origin/develop' into pgrete/pmd-output

6a3a80d

Fix handling of output variable names

28d725d

Fix reading chunks for sparsely populated output files

a039ea1

Merge remote-tracking branch 'origin/develop' into pgrete/pmd-output

ae8519f

Dont tell anyone I spent days on this...

7476641

Temp disable dumping Views from device

39d4b99

Merge branch 'develop' into pgrete/pmd-output

d2ba882

Merge branch 'pgrete/fix-hasghost-restart' into pgrete/pmd-output

626303d

Dump deref cnt in opmd restart

4241198

Merge branch 'develop' into pgrete/pmd-output

101ebf2

install openpmd in macOS CI

60a38b2

Merge branch 'develop' into pgrete/pmd-output

461eeaa

Merge remote-tracking branch 'origin/develop' into pgrete/pmd-output

cd007d3

Remove extraneous popRegion

28020db

Merge remote-tracking branch 'origin/develop' into pgrete/pmd-output

5324e00

Fix formatting

af4b966

pgrete added 7 commits July 10, 2026 14:09

Fix single precision compilation with Real=float

013f360

Add missed conversions

e922a69

Merge remote-tracking branch 'origin/pgrete/single-prec-compile' into…

1de0f87

… lkasselm/fft

Bump Heffte version to latest and ensure it work as external lib and …

2a8fb80

…build-in

Fix precision. Add heffte option. Move to namespace. Add CalcSpec poi…

31d72f4

…nter interface.

Fix copyright

6138245

Add check for CalcSpec fun

3b6c778

pgrete approved these changes Jul 10, 2026

View reviewed changes

pgrete changed the base branch from develop to pgrete/single-prec-compile July 10, 2026 17:36

pgrete requested a review from Yurlungur July 10, 2026 17:36

Yurlungur approved these changes Jul 10, 2026

View reviewed changes

Base automatically changed from pgrete/single-prec-compile to develop July 16, 2026 13:08

Merge remote-tracking branch 'origin/develop' into lkasselm/fft

6b2b170

pgrete enabled auto-merge (squash) July 16, 2026 13:51

pgrete added 2 commits July 16, 2026 18:38

Bump container and dep to rocm7.2.4

098fd66

Merge branch 'pgrete/bump-rocm-container' into lkasselm/fft

023d4cf

pgrete mentioned this pull request Jul 16, 2026

[Infrastructure/trivial] Bump Rocm CI container to 7.2.4 #1414

Merged

13 tasks

Merge remote-tracking branch 'origin/develop' into lkasselm/fft

5beaaa5

Enable FFT only with MPI

45e2176

pgrete changed the title ~~Fourier Transforms~~ WIP Fourier Transforms Jul 29, 2026

pgrete changed the title ~~WIP Fourier Transforms~~ Fourier Transforms Jul 29, 2026

pgrete disabled auto-merge July 29, 2026 08:10

pgrete enabled auto-merge (squash) July 29, 2026 08:11

pgrete merged commit d9d4af6 into develop Jul 29, 2026
37 of 38 checks passed

pgrete deleted the lkasselm/fft branch July 29, 2026 13:26

		fout << std::format("{:d} {:.15e} {:.15e} {:.15e}\n", i, spectra_h(i, 0),
		spectra_h(i, 1), spectra_h(i, 2));

Uh oh!

Conversation

lkasselm commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

pgrete commented Jul 10, 2026

Uh oh!

Yurlungur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgrete commented Jul 16, 2026

Uh oh!

pgrete commented Jul 16, 2026

Uh oh!

pgrete commented Jul 28, 2026

Uh oh!

Uh oh!

pgrete commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lkasselm commented Jun 4, 2026 •

edited

Loading